Machine Learning review and intro to tidymodels
Read through and follow along with the Machine Learning review with an intro to the tidymodels package posted on the Course Materials page.
Tasks:
- Read about the hotel booking data,
hotels, on the Tidy Tuesday page it came from. There is also a link to an article from the original authors. The outcome we will be predicting is called is_canceled.
- Without doing any analysis, what are some variables you think might be predictive and why?
_ What are some problems that might exist with the data? You might think about how it was collected and who did the collecting.
- If we construct a model, what type of conclusions will be able to draw from it?
Some variables that might be predictive are previous cancellations - High number of previous cancellations could mean a higher chance of cancellation previous_bookings_not_canceled - High number of no cancellations wculd mean a lower chance of cancellation deposit_type - Non Refund type deposits would probably be less likely to be cancelled than No Deposit or Refundable type deposits booking_changes - High number of booking changes could mean the stay is more prone to changes and so, cancellation lead_time - Stays booked a longer time in advance could be more likely to be cancelled than stays booked closer to the day of the stay.
Some issues with the data are possible privacy problems of hotel guests.
We would be able to predict which bookings are more likely to get canceled.
- Create some exploratory plots or table summaries of the data, concentrating most on relationships with the response variable. Keep in mind the response variable is numeric, 0 or 1. You may want to make it categorical (you also may not). Be sure to also examine missing values or other interesting values.
# deposit_type vs is_canceled
hotels %>%
count(deposit_type)
hotels %>%
group_by(deposit_type) %>%
summarise( tot_cancel = sum(is_canceled), tot_books = n() ) %>%
mutate(perc_cancel = tot_cancel/tot_books)
what?
#previous_cancellations vs is_canceled
hotels %>%
mutate(total_previous = previous_cancellations+previous_bookings_not_canceled) %>%
filter(total_previous != 0) %>%
mutate(prev_cancel_perc = previous_cancellations/total_previous) %>%
ggplot(aes(x = prev_cancel_perc, fill = (is_canceled==1))) +
geom_density(aes(alpha = 0.2))

#lead time vs is_canceled
hotels %>%
ggplot(aes(x = lead_time, fill =(is_canceled==1)))+
geom_density(aes(alpha= 0.2))

- First, we will do a couple things to get the data ready, including making the outcome a factor (needs to be that way for logistic regression), removing the year variable and some reservation status variables, and removing missing values (not NULLs but true missing values). Split the data into a training and test set, stratifying on the outcome variable,
is_canceled. Since we have a lot of data, we’re going to split the data 50/50 between training and test. I have already set.seed() for you. Be sure to use hotels_mod in the splitting.
hotels_mod <- hotels %>%
mutate(is_canceled = as.factor(is_canceled)) %>%
mutate(across(where(is.character), as.factor)) %>%
select(-arrival_date_year,
-reservation_status,
-reservation_status_date) %>%
add_n_miss() %>%
filter(n_miss_all == 0) %>%
select(-n_miss_all)
hotels_mod
set.seed(494)
hotels_split <- initial_split(hotels_mod,
prop = .75)
hotels_training<-training(hotels_split)
hotels_testing<- testing(hotels_split)
- In this next step, we are going to do the pre-processing. Usually, I won’t tell you exactly what to do here, but for your first exercise, I’ll tell you the steps.
- Set up the recipe with
is_canceled as the outcome and all other variables as predictors (HINT: ~.).
- Use a
step_XXX() function or functions (I think there are other ways to do this, but I found step_mutate_at() easiest) to create some indicator variables for the following variables:children,babies, andprevious_cancellations`. So, the new variable should be a 1 if the original is more than 0 and 0 otherwise. Make sure you do this in a way that accounts for values that may be larger than any we see in the dataset.
- For the
agent and company variables, make new indicator variables that are 1 if they have a value of NULL and 0 otherwise.
- Use
fct_lump_n() to lump together countries that aren’t in the top 5 most occurring.
- If you used new names for some of the new variables you created, then remove any variables that are no longer needed.
- Use
step_normalize() to center and scale all the non-categorical predictor variables. (Do this BEFORE creating dummy variables. When I tried to do it after, I ran into an error - I’m still investigating why.)
- Create dummy variables for all factors/categorical predictor variables (make sure you have
-all_outcomes() in this part!!).
- Use the
prep() and juice() functions to apply the steps to the training data just to check that everything went as planned.
#alternative
hotels_recipe <- recipe(is_canceled ~ .,
data = hotels_training)
hotels_mod
hotels_recipe <- hotels_recipe %>%
step_mutate(children, fn = as.factor(children> 0),
babies, fn = as.factor(babies> 0),
previous_cancellations, fn = as.factor(previous_cancellations> 0)) %>%
step_mutate(agent = as.numeric(agent == "NULL"),
company = as.numeric(company == "NULL")) %>%
step_mutate(country = fct_lump_n(f = (country),5)) %>%
step_normalize(all_predictors(),
-all_nominal()) %>%
step_dummy(all_nominal(),-all_outcomes())
hotels_recipe %>%
prep(hotels_training) %>%
juice()
- In this step we will set up a LASSO model and workflow.
- In general, why would we want to use LASSO instead of regular logistic regression? (HINT: think about what happens to the coefficients).
- Define the model type, set the engine, set the
penalty argument to tune() as a placeholder, and set the mode.
- Create a workflow with the recipe and model.
#Defining model type
hotels_mod <- logistic_reg(penalty =tune()) %>%
set_engine("glmnet") %>%
set_mode("classification")
#Defining workflow
hotels_wf <- workflow() %>%
add_recipe(hotels_recipe) %>%
add_model(hotels_mod)
- In this step, we’ll tune the model and fit the model using the best tuning parameter to the entire training dataset.
- Create a 5-fold cross-validation sample. We’ll use this later. I have set the seed for you.
- Use the
grid_regular() function to create a grid of 10 potential penalty parameters (we’re keeping this sort of small because the dataset is pretty large). Use that with the 5-fold cv data to tune the model.
- Use the
tune_grid() function to fit the models with different tuning parameters to the different cross-validation sets.
- Use the
collect_metrics() function to collect all the metrics from the previous step and create a plot with the accuracy on the y-axis and the penalty term on the x-axis. Put the x-axis on the log scale.
- Use the
select_best() function to find the best tuning parameter, fit the model using that tuning parameter to the entire training set (HINT: finalize_workflow() and fit()), and display the model results using pull_workflow_fit() and tidy(). Are there some variables with coefficients of 0?
set.seed(494) # for reproducibility
hotels_cv <- vfold_cv(hotels_training, v = 5)
pen_grid <- grid_regular(penalty(),levels = 10)#needs a list of 10 penalty parameteres
pen_grid
# hotels_fit_cv <-
# # Tell it the workflow
# hotels_wf %>%
# # Fit the model (using the workflow) to the cv data
# fit_resamples(hotels_cv)
hotels_lasso_tune <-
hotels_wf %>%
tune_grid(hotels_recipe, resamples = hotels_cv, grid = pen_grid) #instead of fit_resamples()
hotels_lasso_tune
# collect_metrics(hotels_res)
collect_metrics(hotels_lasso_tune)
collect_metrics(hotels_lasso_tune) %>%
ggplot(aes(x = log10(penalty), y= mean, color = .metric))+
geom_point()

hotels_bestparam<- select_best(hotels_lasso_tune, metric = 'roc_auc')
hotels_bestparam
hotels_fin_wf <- hotels_wf %>%
finalize_workflow(hotels_bestparam)
hotels_fin_wf
## == Workflow ====================================================================
## Preprocessor: Recipe
## Model: logistic_reg()
##
## -- Preprocessor ----------------------------------------------------------------
## 5 Recipe Steps
##
## * step_mutate()
## * step_mutate()
## * step_mutate()
## * step_normalize()
## * step_dummy()
##
## -- Model -----------------------------------------------------------------------
## Logistic Regression Model Specification (classification)
##
## Main Arguments:
## penalty = 1e-10
##
## Computational engine: glmnet
hotels_fit <- hotels_fin_wf %>%
fit(data = hotels_training)
#hotels_fit
# #alternative
# set.seed(494) # for reproducibility
# hotels_cv <- vfold_cv(hotels_training, v = 5)
# pen_grid <- grid_regular(penalty(),levels = 10)#needs a list of 10 penalty parameteres
#
#
# hotels_fit <- hotels_wf %>%
# add_model(hotels_mod) %>%
# fit(data = hotels_training)
#
# hotels_fit %>%
# pull_workflow_fit() %>%
# tidy()
- Now that we have a model, let’s evaluate it a bit more. All we have looked at so far is the cross-validated accuracy from the previous step.
- Create a variable importance graph. Which variables show up as the most important? Are you surprised?
- Use the
last_fit() function to fit the final model and then apply it to the testing data. Report the metrics from the testing data using the collet_metrics() function. How do they compare to the cross-validated metrics?
- Use the
collect_predictions() function to find the predicted probabilities and classes for the test data. Save this to a new dataset called preds. Then, use the conf_mat() function from dials (part of tidymodels) to create a confusion matrix showing the predicted classes vs. the true classes. What is the true positive rate (sensitivity)? What is the true negative rate (specificity)? See this Wikipedia reference if you (like me) tend to forget these definitions.
- Use the
preds dataset you just created to create a density plot of the predicted probabilities of canceling (the variable is called .pred_1), filling by is_canceled. Use an alpha = .5 and color = NA in the geom_density(). Answer these questions: a. What would this graph look like for a model with an accuracy that was close to 1? b. Our predictions are classified as canceled if their predicted probability of canceling is greater than .5. If we wanted to have a high true positive rate, should we make the cutoff for predicted as canceled higher or lower than .5? c. What happens to the true negative rate if we try to get a higher true positive rate?
library(vip)
hotels_fit %>%
pull_workflow_fit() %>%
vip()
reserved_room_type, deposit_type and assigned_room_type appear to be key predictor variables. I expected deposit_type to be an important predictor but the other 2 predictors which are associated with room_type was suprising to me.
hotels_fin_test <- hotels_fin_wf %>%
last_fit(hotels_split)
hotels_fin_test %>% collect_metrics()
The cross validated metrics were roc_auc:0.894 and accuracy: 0.814 and compared to that the testing metrics are about the same with the accuracy value very slightly better than the cross validated one.
preds<-collect_predictions(hotels_fin_test)
hotels_mat<-preds%>%
conf_mat(is_canceled, .pred_class)
hotels_mat
## Truth
## Prediction 0 1
## 0 17083 3907
## 1 1614 7242
true_pos <- 7007
true_neg <- 17324
false_pos<- 1614
false_neg<- 3901
sensitivity<- true_pos/(true_pos+false_neg)
specificity<- true_neg/(true_neg+false_pos)
sensitivity
## [1] 0.6423726
specificity
## [1] 0.9147745
preds%>%
ggplot(aes(x = .pred_1, fill = is_canceled))+
geom_density(alpha = 0.5, color = NA)

We would have 2 peaks at 0 and 1 with 1 corresponding and the middle parts would be close to 0.
We lower the cutoff because the specificity is higher than sensitivity
It will go down, since we will be making more is_cancelled = true predictions. False positives would increase and so, true negatives would increase.
- Let’s say that this model is going to be applied to bookings 14 days in advance of their arrival at each hotel, and someone who works for the hotel will make a phone call to the person who made the booking. During this phone call, they will try to assure that the person will be keeping their reservation or that they will be canceling in which case they can do that now and still have time to fill the room. How should the hotel go about deciding who to call? How could they measure whether it was worth the effort to do the calling? Can you think of another way they might use the model?
By the variable importance plot we did earlier, the hotel should probably call people who have reserved room type p, made non refundable deposits or assigned room type I (in order of importance) since they are more likely to be someone who is willing to cancel.
To measure if it was worth the effort of calling you could maybe develop a metric which involves the probability of cancellation of that booking, the price of the room (how much the hotel could possibly saving by minimizing cancellations) and the waiting list of the hotel.
They might be able to use the model to do overbooking similar to airlines but this could be more problematic since there might not be a large supply of alternative rooms in the area similar to a large supply of alternative flights that airlines could provide.
- How might you go about questioning and evaluating the model in terms of fairness? Are there any questions you would like to ask of the people who collected the data?
Country bias could be something to consider since that is a variable in the dataset. The model could give questionable results due to issues such as questionable sample sizes from certain countries and this might lead to some sort of discriminatory practice.
You would have to make sure that the calling to cancel bookings do not have socioeconomic or racial bias hidden in variables such as reserved_room_type and deposit_type.
LS0tDQp0aXRsZTogJ0Fzc2lnbm1lbnQgIzEnDQpuYW1lOiAnTmlrZXRoIEdhbWFnZScNCm91dHB1dDogDQogIGh0bWxfZG9jdW1lbnQ6DQogICAgdG9jOiB0cnVlDQogICAgdG9jX2Zsb2F0OiB0cnVlDQogICAgZGZfcHJpbnQ6IHBhZ2VkDQogICAgY29kZV9kb3dubG9hZDogdHJ1ZQ0KLS0tDQoNCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFfQ0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KGVjaG8gPSBUUlVFLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFKQ0KYGBgDQoNCmBgYHtyIGxpYnJhcmllc30NCmxpYnJhcnkodGlkeXZlcnNlKSAgICAgICAgICMgZm9yIGdyYXBoaW5nIGFuZCBkYXRhIGNsZWFuaW5nDQpsaWJyYXJ5KHRpZHltb2RlbHMpICAgICAgICAjIGZvciBtb2RlbGluZw0KbGlicmFyeShuYW5pYXIpICAgICAgICAgICAgIyBmb3IgYW5hbHl6aW5nIG1pc3NpbmcgdmFsdWVzDQpsaWJyYXJ5KHZpcCkgICAgICAgICAgICAgICAjIGZvciB2YXJpYWJsZSBpbXBvcnRhbmNlIHBsb3RzDQp0aGVtZV9zZXQodGhlbWVfbWluaW1hbCgpKSAjIExpc2EncyBmYXZvcml0ZSB0aGVtZQ0KYGBgDQoNCmBgYHtyIGRhdGF9DQpob3RlbHMgPC0gcmVhZHI6OnJlYWRfY3N2KCdodHRwczovL3Jhdy5naXRodWJ1c2VyY29udGVudC5jb20vcmZvcmRhdGFzY2llbmNlL3RpZHl0dWVzZGF5L21hc3Rlci9kYXRhLzIwMjAvMjAyMC0wMi0xMS9ob3RlbHMuY3N2JykNCmBgYA0KDQoNCldoZW4geW91IGZpbmlzaCB0aGUgYXNzaWdubWVudCwgcmVtb3ZlIHRoZSBgI2AgZnJvbSB0aGUgb3B0aW9ucyBjaHVuayBhdCB0aGUgdG9wLCBzbyB0aGF0IG1lc3NhZ2VzIGFuZCB3YXJuaW5ncyBhcmVuJ3QgcHJpbnRlZC4gSWYgeW91IGFyZSBnZXR0aW5nIGVycm9ycyBpbiB5b3VyIGNvZGUsIGFkZCBgZXJyb3IgPSBUUlVFYCBzbyB0aGF0IHRoZSBmaWxlIGtuaXRzLiBJIHdvdWxkIHJlY29tbWVuZCBub3QgcmVtb3ZpbmcgdGhlIGAjYCB1bnRpbCB5b3UgYXJlIGNvbXBsZXRlbHkgZmluaXNoZWQuDQoNCiMjIFNldHRpbmcgdXAgR2l0IGFuZCBHaXRIdWIgaW4gUlN0dWRpbw0KDQpSZWFkIHRoZSBbUXVpY2sgSW50cm9dKGh0dHBzOi8vYWR2YW5jZWQtZHMtaW4tci5uZXRsaWZ5LmFwcC9wb3N0cy8yMDIxLTAxLTI4LWdpdGdpdGh1Yi8jcXVpY2staW50cm8pIHNlY3Rpb24gb2YgdGhlIFVzaW5nIGdpdCBhbmQgR2l0SHViIGluIFIgU3R1ZGlvIHNldCBvZiBDb3Vyc2UgTWF0ZXJpYWxzLiBTZXQgdXAgR2l0IGFuZCBHaXRIdWIgYW5kIGNyZWF0ZSBhIEdpdEh1YiByZXBvIGFuZCBhc3NvY2lhdGVkIFIgUHJvamVjdCAoZG9uZSBmb3IgeW91IHdoZW4geW91IGNsb25lIHRoZSByZXBvKSBmb3IgdGhpcyBob21ld29yayBhc3NpZ25tZW50LiBQdXQgdGhpcyBmaWxlIGludG8gdGhlIHByb2plY3QuIFlvdSBzaG91bGQgYWx3YXlzIG9wZW4gdGhlIFIgUHJvamVjdCAoLlJwcm9qKSBmaWxlIHdoZW4geW91IHdvcmsgd2l0aCBhbnkgb2YgdGhlIGZpbGVzIGluIHRoZSBwcm9qZWN0LiANCg0KKipUYXNrKio6IEJlbG93LCBwb3N0IGEgbGluayB0byB5b3VyIEdpdEh1YiByZXBvc2l0b3J5Lg0KDQoNCiMjIENyZWF0aW5nIGEgd2Vic2l0ZQ0KDQpZb3UnbGwgYmUgdXNpbmcgUlN0dWRpbyB0byBjcmVhdGUgYSBwZXJzb25hbCB3ZWJzaXRlIHRvIHNob3djYXNlIHlvdXIgd29yayBmcm9tIHRoaXMgY2xhc3MhIFN0YXJ0IGJ5IHdhdGNoaW5nIHRoZSBbU2hhcmluZyBvbiBTaG9ydCBOb3RpY2VdKGh0dHBzOi8vcnN0dWRpby5jb20vcmVzb3VyY2VzL3dlYmluYXJzL3NoYXJpbmctb24tc2hvcnQtbm90aWNlLWhvdy10by1nZXQteW91ci1tYXRlcmlhbHMtb25saW5lLXdpdGgtci1tYXJrZG93bi8pIHdlYmluYXIgYnkgQWxpc29uIEhpbGwgYW5kIERlc2lyw6llIERlIExlb24gb2YgUlN0dWRpby4gVGhpcyBzaG91bGQgaGVscCB5b3UgY2hvb3NlIHRoZSB0eXBlIG9mIHdlYnNpdGUgeW91J2QgbGlrZSB0byBjcmVhdGUuIA0KDQpPbmNlIHlvdSd2ZSBjaG9zZW4gdGhhdCwgeW91IG1pZ2h0IHdhbnQgdG8gbG9vayB0aHJvdWdoIHNvbWUgb2YgdGhlIG90aGVyICpCdWlsZGluZyBhIHdlYnNpdGUqIHJlc291cmNlcyBJIHBvc3RlZCBvbiB0aGUgW3Jlc291cmNlcyBwYWdlXShodHRwczovL2FkdmFuY2VkLWRzLWluLXIubmV0bGlmeS5hcHAvcmVzb3VyY2VzLmh0bWwpIG9mIG91ciBjb3Vyc2Ugd2Vic2l0ZS4gSSBoaWdobHkgcmVjb21tZW5kIG1ha2luZyBhIG5pY2UgbGFuZGluZyBwYWdlIHdoZXJlIHlvdSBnaXZlIGEgYnJpZWYgaW50cm9kdWN0aW9uIG9mIHlvdXJzZWxmLiANCg0KKipUYXNrcyoqOg0KDQoqIEluY2x1ZGUgYSBsaW5rIHRvIHlvdXIgd2Vic2l0ZSBiZWxvdy4gKElmIGFueW9uZSBkb2VzIG5vdCB3YW50IHRvIHBvc3QgYSB3ZWJzaXRlIHB1YmxpY2x5LCBwbGVhc2UgdGFsayB0byBtZSBhbmQgd2Ugd2lsbCBmaW5kIGEgZGlmZmVyZW50IHNvbHV0aW9uKS4gIA0KDQpodHRwczovL2RhenpsaW5nLWFsbGVuLTU0MzBlMi5uZXRsaWZ5LmFwcC8oV2FudCB0byBtYWtlIGEgYmxvZ2Rvd24gc2l0ZSBidXQgbWFkZSB0aGlzIGZvciBub3cpDQoNCiogTGlzdGVuIHRvIGF0IGxlYXN0IHRoZSBmaXJzdCAyMCBtaW51dGVzIG9mICJCdWlsZGluZyBhIENhcmVlciBpbiBEYXRhIFNjaWVuY2UsIENoYXB0ZXIgNDogQnVpbGRpbmcgYSBQb3J0Zm9saW8iLiBHbyB0byB0aGUgbWFpbiBbcG9kY2FzdCB3ZWJzaXRlXShodHRwczovL3BvZGNhc3QuYmVzdGJvb2suY29vbC8pIGFuZCBuYXZpZ2F0ZSB0byBhIHBvZGNhc3QgcHJvdmlkZXIgdGhhdCB3b3JrcyBmb3IgeW91IHRvIGZpbmQgdGhhdCBzcGVjaWZpYyBlcGlzb2RlLiBXcml0ZSAyLTMgc2VudGVuY2VzIHJlZmxlY3Rpbmcgb24gd2hhdCB0aGV5IGRpc2N1c3NlZCBhbmQgd2h5IGNyZWF0aW5nIGEgd2Vic2l0ZSBtaWdodCBiZSBoZWxwZnVsIGZvciB5b3UuICANCg0KSXQncyBhIGdvb2Qgd2F5IHRvIGRpc3BsYXkgeW91ciB3b3JrLCBlc3BlY2lhbGx5IHdoZW4gaW50ZXJ2aWV3aW5nIGFuZCBpdCBoZWxwcyBvcmdhbml6ZSB5b3VyIHdvcmsgZm9yIHlvdXJzZWxmLiANCg0KKiAoT3B0aW9uYWwpIENyZWF0ZSBhbiBSIHBhY2thZ2Ugd2l0aCB5b3VyIG93biBjdXN0b21pemVkIGBncHBsb3QyYCB0aGVtZSEgV3JpdGUgYSBwb3N0IG9uIHlvdXIgd2Vic2l0ZSBhYm91dCB3aHkgeW91IG1hZGUgdGhlIGNob2ljZXMgeW91IGRpZCBmb3IgdGhlIHRoZW1lLiBTZWUgdGhlICpCdWlsZGluZyBhbiBSIHBhY2thZ2UqIGFuZCAqQ3VzdG9tIGBnZ3Bsb3QyYCB0aGVtZXMqIFtyZXNvdXJjZXNdKGh0dHBzOi8vYWR2YW5jZWQtZHMtaW4tci5uZXRsaWZ5LmFwcC9yZXNvdXJjZXMuaHRtbCkuIA0KDQojIyBNYWNoaW5lIExlYXJuaW5nIHJldmlldyBhbmQgaW50cm8gdG8gYHRpZHltb2RlbHNgDQoNClJlYWQgdGhyb3VnaCBhbmQgZm9sbG93IGFsb25nIHdpdGggdGhlIFtNYWNoaW5lIExlYXJuaW5nIHJldmlldyB3aXRoIGFuIGludHJvIHRvIHRoZSBgdGlkeW1vZGVsc2AgcGFja2FnZV0oaHR0cHM6Ly9hZHZhbmNlZC1kcy1pbi1yLm5ldGxpZnkuYXBwL3Bvc3RzLzIwMjEtMDMtMTYtbWwtcmV2aWV3LykgcG9zdGVkIG9uIHRoZSBDb3Vyc2UgTWF0ZXJpYWxzIHBhZ2UuIA0KDQoqKlRhc2tzKio6DQoNCjEuIFJlYWQgYWJvdXQgdGhlIGhvdGVsIGJvb2tpbmcgZGF0YSwgYGhvdGVsc2AsIG9uIHRoZSBbVGlkeSBUdWVzZGF5IHBhZ2VdKGh0dHBzOi8vZ2l0aHViLmNvbS9yZm9yZGF0YXNjaWVuY2UvdGlkeXR1ZXNkYXkvYmxvYi9tYXN0ZXIvZGF0YS8yMDIwLzIwMjAtMDItMTEvcmVhZG1lLm1kKSBpdCBjYW1lIGZyb20uIFRoZXJlIGlzIGFsc28gYSBsaW5rIHRvIGFuIGFydGljbGUgZnJvbSB0aGUgb3JpZ2luYWwgYXV0aG9ycy4gVGhlIG91dGNvbWUgd2Ugd2lsbCBiZSBwcmVkaWN0aW5nIGlzIGNhbGxlZCBgaXNfY2FuY2VsZWRgLiANCiAgLSBXaXRob3V0IGRvaW5nIGFueSBhbmFseXNpcywgd2hhdCBhcmUgc29tZSB2YXJpYWJsZXMgeW91IHRoaW5rIG1pZ2h0IGJlIHByZWRpY3RpdmUgYW5kIHdoeT8gIA0KICBfIFdoYXQgYXJlIHNvbWUgcHJvYmxlbXMgdGhhdCBtaWdodCBleGlzdCB3aXRoIHRoZSBkYXRhPyBZb3UgbWlnaHQgdGhpbmsgYWJvdXQgaG93IGl0IHdhcyBjb2xsZWN0ZWQgYW5kIHdobyBkaWQgdGhlIGNvbGxlY3RpbmcuICANCiAgLSBJZiB3ZSBjb25zdHJ1Y3QgYSBtb2RlbCwgd2hhdCB0eXBlIG9mIGNvbmNsdXNpb25zIHdpbGwgYmUgYWJsZSB0byBkcmF3IGZyb20gaXQ/ICANCiAgDQogIFNvbWUgdmFyaWFibGVzIHRoYXQgbWlnaHQgYmUgcHJlZGljdGl2ZSBhcmUgDQogIHByZXZpb3VzIGNhbmNlbGxhdGlvbnMgLSBIaWdoIG51bWJlciBvZiBwcmV2aW91cyBjYW5jZWxsYXRpb25zIGNvdWxkIG1lYW4gYSBoaWdoZXIgY2hhbmNlIG9mIGNhbmNlbGxhdGlvbg0KICBwcmV2aW91c19ib29raW5nc19ub3RfY2FuY2VsZWQgLSBIaWdoIG51bWJlciBvZiBubyBjYW5jZWxsYXRpb25zIHdjdWxkIG1lYW4gYSBsb3dlciBjaGFuY2Ugb2YgY2FuY2VsbGF0aW9uDQogIGRlcG9zaXRfdHlwZSAtIE5vbiBSZWZ1bmQgdHlwZSBkZXBvc2l0cyB3b3VsZCBwcm9iYWJseSBiZSBsZXNzIGxpa2VseSB0byBiZSBjYW5jZWxsZWQgdGhhbiBObyBEZXBvc2l0IG9yIFJlZnVuZGFibGUgdHlwZSBkZXBvc2l0cw0KICBib29raW5nX2NoYW5nZXMgLSBIaWdoIG51bWJlciBvZiBib29raW5nIGNoYW5nZXMgY291bGQgbWVhbiB0aGUgc3RheSBpcyBtb3JlIHByb25lIHRvIGNoYW5nZXMgYW5kIHNvLCBjYW5jZWxsYXRpb24NCiAgbGVhZF90aW1lIC0gU3RheXMgYm9va2VkIGEgbG9uZ2VyIHRpbWUgaW4gYWR2YW5jZSBjb3VsZCBiZSBtb3JlIGxpa2VseSB0byBiZSBjYW5jZWxsZWQgdGhhbiBzdGF5cyBib29rZWQgY2xvc2VyIHRvIHRoZSBkYXkgb2YgdGhlIHN0YXkuDQogIA0KICBTb21lIGlzc3VlcyB3aXRoIHRoZSBkYXRhIGFyZSBwb3NzaWJsZSBwcml2YWN5IHByb2JsZW1zIG9mIGhvdGVsIGd1ZXN0cy4NCiAgDQogIFdlIHdvdWxkIGJlIGFibGUgdG8gcHJlZGljdCB3aGljaCBib29raW5ncyBhcmUgbW9yZSBsaWtlbHkgdG8gZ2V0IGNhbmNlbGVkLg0KICANCjIuIENyZWF0ZSBzb21lIGV4cGxvcmF0b3J5IHBsb3RzIG9yIHRhYmxlIHN1bW1hcmllcyBvZiB0aGUgZGF0YSwgY29uY2VudHJhdGluZyBtb3N0IG9uIHJlbGF0aW9uc2hpcHMgd2l0aCB0aGUgcmVzcG9uc2UgdmFyaWFibGUuIEtlZXAgaW4gbWluZCB0aGUgcmVzcG9uc2UgdmFyaWFibGUgaXMgbnVtZXJpYywgMCBvciAxLiBZb3UgbWF5IHdhbnQgdG8gbWFrZSBpdCBjYXRlZ29yaWNhbCAoeW91IGFsc28gbWF5IG5vdCkuIEJlIHN1cmUgdG8gYWxzbyBleGFtaW5lIG1pc3NpbmcgdmFsdWVzIG9yIG90aGVyIGludGVyZXN0aW5nIHZhbHVlcy4gIA0KDQpgYGB7cn0NCiMgZGVwb3NpdF90eXBlIHZzIGlzX2NhbmNlbGVkDQpob3RlbHMgJT4lIA0KICBjb3VudChkZXBvc2l0X3R5cGUpDQpob3RlbHMgJT4lIA0KICBncm91cF9ieShkZXBvc2l0X3R5cGUpICU+JSANCiAgc3VtbWFyaXNlKCB0b3RfY2FuY2VsID0gc3VtKGlzX2NhbmNlbGVkKSwgdG90X2Jvb2tzID0gbigpICkgJT4lIA0KICBtdXRhdGUocGVyY19jYW5jZWwgPSB0b3RfY2FuY2VsL3RvdF9ib29rcykNCmBgYA0Kd2hhdD8NCg0KYGBge3IgfQ0KI3ByZXZpb3VzX2NhbmNlbGxhdGlvbnMgdnMgaXNfY2FuY2VsZWQNCmhvdGVscyAlPiUgDQogIG11dGF0ZSh0b3RhbF9wcmV2aW91cyA9IHByZXZpb3VzX2NhbmNlbGxhdGlvbnMrcHJldmlvdXNfYm9va2luZ3Nfbm90X2NhbmNlbGVkKSAlPiUgDQogIGZpbHRlcih0b3RhbF9wcmV2aW91cyAhPSAwKSAlPiUgDQogIG11dGF0ZShwcmV2X2NhbmNlbF9wZXJjID0gcHJldmlvdXNfY2FuY2VsbGF0aW9ucy90b3RhbF9wcmV2aW91cykgJT4lIA0KICBnZ3Bsb3QoYWVzKHggPSBwcmV2X2NhbmNlbF9wZXJjLCBmaWxsID0gKGlzX2NhbmNlbGVkPT0xKSkpICsNCiAgZ2VvbV9kZW5zaXR5KGFlcyhhbHBoYSA9IDAuMikpDQogIA0KICANCg0KDQpgYGANCg0KYGBge3J9DQojbGVhZCB0aW1lIHZzIGlzX2NhbmNlbGVkDQpob3RlbHMgJT4lIA0KICBnZ3Bsb3QoYWVzKHggPSBsZWFkX3RpbWUsIGZpbGwgPShpc19jYW5jZWxlZD09MSkpKSsNCiAgZ2VvbV9kZW5zaXR5KGFlcyhhbHBoYT0gMC4yKSkNCg0KYGBgDQoNCjMuIEZpcnN0LCB3ZSB3aWxsIGRvIGEgY291cGxlIHRoaW5ncyB0byBnZXQgdGhlIGRhdGEgcmVhZHksIGluY2x1ZGluZyBtYWtpbmcgdGhlIG91dGNvbWUgYSBmYWN0b3IgKG5lZWRzIHRvIGJlIHRoYXQgd2F5IGZvciBsb2dpc3RpYyByZWdyZXNzaW9uKSwgcmVtb3ZpbmcgdGhlIHllYXIgdmFyaWFibGUgYW5kIHNvbWUgcmVzZXJ2YXRpb24gc3RhdHVzIHZhcmlhYmxlcywgYW5kIHJlbW92aW5nIG1pc3NpbmcgdmFsdWVzIChub3QgTlVMTHMgYnV0IHRydWUgbWlzc2luZyB2YWx1ZXMpLiBTcGxpdCB0aGUgZGF0YSBpbnRvIGEgdHJhaW5pbmcgYW5kIHRlc3Qgc2V0LCBzdHJhdGlmeWluZyBvbiB0aGUgb3V0Y29tZSB2YXJpYWJsZSwgYGlzX2NhbmNlbGVkYC4gU2luY2Ugd2UgaGF2ZSBhIGxvdCBvZiBkYXRhLCB3ZSdyZSBnb2luZyB0byBzcGxpdCB0aGUgZGF0YSA1MC81MCBiZXR3ZWVuIHRyYWluaW5nIGFuZCB0ZXN0LiBJIGhhdmUgYWxyZWFkeSBgc2V0LnNlZWQoKWAgZm9yIHlvdS4gQmUgc3VyZSB0byB1c2UgYGhvdGVsc19tb2RgIGluIHRoZSBzcGxpdHRpbmcuDQoNCmBgYHtyfQ0KaG90ZWxzX21vZCA8LSBob3RlbHMgJT4lIA0KICBtdXRhdGUoaXNfY2FuY2VsZWQgPSBhcy5mYWN0b3IoaXNfY2FuY2VsZWQpKSAlPiUgDQogIG11dGF0ZShhY3Jvc3Mod2hlcmUoaXMuY2hhcmFjdGVyKSwgYXMuZmFjdG9yKSkgJT4lIA0KICBzZWxlY3QoLWFycml2YWxfZGF0ZV95ZWFyLA0KICAgICAgICAgLXJlc2VydmF0aW9uX3N0YXR1cywNCiAgICAgICAgIC1yZXNlcnZhdGlvbl9zdGF0dXNfZGF0ZSkgJT4lIA0KICBhZGRfbl9taXNzKCkgJT4lIA0KICBmaWx0ZXIobl9taXNzX2FsbCA9PSAwKSAlPiUgDQogIHNlbGVjdCgtbl9taXNzX2FsbCkNCg0KaG90ZWxzX21vZA0Kc2V0LnNlZWQoNDk0KQ0KDQpob3RlbHNfc3BsaXQgPC0gaW5pdGlhbF9zcGxpdChob3RlbHNfbW9kLCANCiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgcHJvcCA9IC43NSkNCg0KaG90ZWxzX3RyYWluaW5nPC10cmFpbmluZyhob3RlbHNfc3BsaXQpDQpob3RlbHNfdGVzdGluZzwtIHRlc3RpbmcoaG90ZWxzX3NwbGl0KQ0KYGBgDQoNCg0KNC4gSW4gdGhpcyBuZXh0IHN0ZXAsIHdlIGFyZSBnb2luZyB0byBkbyB0aGUgcHJlLXByb2Nlc3NpbmcuIFVzdWFsbHksIEkgd29uJ3QgdGVsbCB5b3UgZXhhY3RseSB3aGF0IHRvIGRvIGhlcmUsIGJ1dCBmb3IgeW91ciBmaXJzdCBleGVyY2lzZSwgSSdsbCB0ZWxsIHlvdSB0aGUgc3RlcHMuIA0KDQoqIFNldCB1cCB0aGUgcmVjaXBlIHdpdGggYGlzX2NhbmNlbGVkYCBhcyB0aGUgb3V0Y29tZSBhbmQgYWxsIG90aGVyIHZhcmlhYmxlcyBhcyBwcmVkaWN0b3JzIChISU5UOiBgfi5gKS4gIA0KKiBVc2UgYSBgc3RlcF9YWFgoKWAgZnVuY3Rpb24gb3IgZnVuY3Rpb25zIChJIHRoaW5rIHRoZXJlIGFyZSBvdGhlciB3YXlzIHRvIGRvIHRoaXMsIGJ1dCBJIGZvdW5kIGBzdGVwX211dGF0ZV9hdCgpYCBlYXNpZXN0KSB0byBjcmVhdGUgc29tZSBpbmRpY2F0b3IgdmFyaWFibGVzIGZvciB0aGUgZm9sYGxvd2luZyB2YXJpYWJsZXM6IGBjaGlsZHJlbmAsIGBiYWJpZXNgLCBhbmQgYHByZXZpb3VzX2NhbmNlbGxhdGlvbnNgLiBTbywgdGhlIG5ldyB2YXJpYWJsZSBzaG91bGQgYmUgYSAxIGlmIHRoZSBvcmlnaW5hbCBpcyBtb3JlIHRoYW4gMCBhbmQgMCBvdGhlcndpc2UuIE1ha2Ugc3VyZSB5b3UgZG8gdGhpcyBpbiBhIHdheSB0aGF0IGFjY291bnRzIGZvciB2YWx1ZXMgdGhhdCBtYXkgYmUgbGFyZ2VyIHRoYW4gYW55IHdlIHNlZSBpbiB0aGUgZGF0YXNldC4gIA0KKiBGb3IgdGhlIGBhZ2VudGAgYW5kIGBjb21wYW55YCB2YXJpYWJsZXMsIG1ha2UgbmV3IGluZGljYXRvciB2YXJpYWJsZXMgdGhhdCBhcmUgMSBpZiB0aGV5IGhhdmUgYSB2YWx1ZSBvZiBgTlVMTGAgYW5kIDAgb3RoZXJ3aXNlLiANCiogVXNlIGBmY3RfbHVtcF9uKClgIHRvIGx1bXAgdG9nZXRoZXIgY291bnRyaWVzIHRoYXQgYXJlbid0IGluIHRoZSB0b3AgNSBtb3N0IG9jY3VycmluZy4gDQoqIElmIHlvdSB1c2VkIG5ldyBuYW1lcyBmb3Igc29tZSBvZiB0aGUgbmV3IHZhcmlhYmxlcyB5b3UgY3JlYXRlZCwgdGhlbiByZW1vdmUgYW55IHZhcmlhYmxlcyB0aGF0IGFyZSBubyBsb25nZXIgbmVlZGVkLiANCiogVXNlIGBzdGVwX25vcm1hbGl6ZSgpYCB0byBjZW50ZXIgYW5kIHNjYWxlIGFsbCB0aGUgbm9uLWNhdGVnb3JpY2FsIHByZWRpY3RvciB2YXJpYWJsZXMuIChEbyB0aGlzIEJFRk9SRSBjcmVhdGluZyBkdW1teSB2YXJpYWJsZXMuIFdoZW4gSSB0cmllZCB0byBkbyBpdCBhZnRlciwgSSByYW4gaW50byBhbiBlcnJvciAtIEknbSBzdGlsbCBpbnZlc3RpZ2F0aW5nIHdoeS4pDQoqIENyZWF0ZSBkdW1teSB2YXJpYWJsZXMgZm9yIGFsbCBmYWN0b3JzL2NhdGVnb3JpY2FsIHByZWRpY3RvciB2YXJpYWJsZXMgKG1ha2Ugc3VyZSB5b3UgaGF2ZSBgLWFsbF9vdXRjb21lcygpYCBpbiB0aGlzIHBhcnQhISkuICANCiogVXNlIHRoZSBgcHJlcCgpYCBhbmQgYGp1aWNlKClgIGZ1bmN0aW9ucyB0byBhcHBseSB0aGUgc3RlcHMgdG8gdGhlIHRyYWluaW5nIGRhdGEganVzdCB0byBjaGVjayB0aGF0IGV2ZXJ5dGhpbmcgd2VudCBhcyBwbGFubmVkLg0KDQoNCg0KYGBge3J9DQojYWx0ZXJuYXRpdmUNCmhvdGVsc19yZWNpcGUgPC0gcmVjaXBlKGlzX2NhbmNlbGVkIH4gLiwgDQogICAgICAgICAgICAgICAgICAgICAgIGRhdGEgPSBob3RlbHNfdHJhaW5pbmcpIA0KDQpob3RlbHNfbW9kDQpob3RlbHNfcmVjaXBlIDwtIGhvdGVsc19yZWNpcGUgJT4lIA0KICBzdGVwX211dGF0ZShjaGlsZHJlbiwgZm4gPSBhcy5mYWN0b3IoY2hpbGRyZW4+IDApLA0KICAgICAgICAgICAgICBiYWJpZXMsIGZuID0gYXMuZmFjdG9yKGJhYmllcz4gMCksDQogICAgICAgICAgICAgIHByZXZpb3VzX2NhbmNlbGxhdGlvbnMsIGZuID0gYXMuZmFjdG9yKHByZXZpb3VzX2NhbmNlbGxhdGlvbnM+IDApKSAlPiUgDQogIHN0ZXBfbXV0YXRlKGFnZW50ID0gYXMubnVtZXJpYyhhZ2VudCA9PSAiTlVMTCIpLA0KICAgICAgICAgICAgICBjb21wYW55ID0gYXMubnVtZXJpYyhjb21wYW55ID09ICJOVUxMIikpICU+JSANCiAgc3RlcF9tdXRhdGUoY291bnRyeSA9IGZjdF9sdW1wX24oZiA9IChjb3VudHJ5KSw1KSkgJT4lIA0KICBzdGVwX25vcm1hbGl6ZShhbGxfcHJlZGljdG9ycygpLCANCiAgICAgICAgICAgICAgICAgLWFsbF9ub21pbmFsKCkpICU+JSANCiAgc3RlcF9kdW1teShhbGxfbm9taW5hbCgpLC1hbGxfb3V0Y29tZXMoKSkNCmBgYA0KDQpgYGB7cn0NCmhvdGVsc19yZWNpcGUgJT4lIA0KICBwcmVwKGhvdGVsc190cmFpbmluZykgJT4lIA0KICBqdWljZSgpDQpgYGANCg0KNS4gSW4gdGhpcyBzdGVwIHdlIHdpbGwgc2V0IHVwIGEgTEFTU08gbW9kZWwgYW5kIHdvcmtmbG93Lg0KDQoqIEluIGdlbmVyYWwsIHdoeSB3b3VsZCB3ZSB3YW50IHRvIHVzZSBMQVNTTyBpbnN0ZWFkIG9mIHJlZ3VsYXIgbG9naXN0aWMgcmVncmVzc2lvbj8gKEhJTlQ6IHRoaW5rIGFib3V0IHdoYXQgaGFwcGVucyB0byB0aGUgY29lZmZpY2llbnRzKS4gIA0KKiBEZWZpbmUgdGhlIG1vZGVsIHR5cGUsIHNldCB0aGUgZW5naW5lLCBzZXQgdGhlIGBwZW5hbHR5YCBhcmd1bWVudCB0byBgdHVuZSgpYCBhcyBhIHBsYWNlaG9sZGVyLCBhbmQgc2V0IHRoZSBtb2RlLiAgDQoqIENyZWF0ZSBhIHdvcmtmbG93IHdpdGggdGhlIHJlY2lwZSBhbmQgbW9kZWwuICANCg0KYGBge3J9DQojRGVmaW5pbmcgbW9kZWwgdHlwZQ0KaG90ZWxzX21vZCA8LSBsb2dpc3RpY19yZWcocGVuYWx0eSA9dHVuZSgpKSAlPiUNCiAgc2V0X2VuZ2luZSgiZ2xtbmV0IikgJT4lIA0KICBzZXRfbW9kZSgiY2xhc3NpZmljYXRpb24iKQ0KDQpgYGANCg0KYGBge3J9DQojRGVmaW5pbmcgd29ya2Zsb3cNCmhvdGVsc193ZiA8LSB3b3JrZmxvdygpICU+JSANCiAgYWRkX3JlY2lwZShob3RlbHNfcmVjaXBlKSAlPiUgDQogIGFkZF9tb2RlbChob3RlbHNfbW9kKQ0KYGBgDQoNCg0KDQo2LiBJbiB0aGlzIHN0ZXAsIHdlJ2xsIHR1bmUgdGhlIG1vZGVsIGFuZCBmaXQgdGhlIG1vZGVsIHVzaW5nIHRoZSBiZXN0IHR1bmluZyBwYXJhbWV0ZXIgdG8gdGhlIGVudGlyZSB0cmFpbmluZyBkYXRhc2V0Lg0KDQoqIENyZWF0ZSBhIDUtZm9sZCBjcm9zcy12YWxpZGF0aW9uIHNhbXBsZS4gV2UnbGwgdXNlIHRoaXMgbGF0ZXIuIEkgaGF2ZSBzZXQgdGhlIHNlZWQgZm9yIHlvdS4gIA0KKiBVc2UgdGhlIGBncmlkX3JlZ3VsYXIoKWAgZnVuY3Rpb24gdG8gY3JlYXRlIGEgZ3JpZCBvZiAxMCBwb3RlbnRpYWwgcGVuYWx0eSBwYXJhbWV0ZXJzICh3ZSdyZSBrZWVwaW5nIHRoaXMgc29ydCBvZiBzbWFsbCBiZWNhdXNlIHRoZSBkYXRhc2V0IGlzIHByZXR0eSBsYXJnZSkuIFVzZSB0aGF0IHdpdGggdGhlIDUtZm9sZCBjdiBkYXRhIHRvIHR1bmUgdGhlIG1vZGVsLiAgDQoqIFVzZSB0aGUgYHR1bmVfZ3JpZCgpYCBmdW5jdGlvbiB0byBmaXQgdGhlIG1vZGVscyB3aXRoIGRpZmZlcmVudCB0dW5pbmcgcGFyYW1ldGVycyB0byB0aGUgZGlmZmVyZW50IGNyb3NzLXZhbGlkYXRpb24gc2V0cy4gIA0KKiBVc2UgdGhlIGBjb2xsZWN0X21ldHJpY3MoKWAgZnVuY3Rpb24gdG8gY29sbGVjdCBhbGwgdGhlIG1ldHJpY3MgZnJvbSB0aGUgcHJldmlvdXMgc3RlcCBhbmQgY3JlYXRlIGEgcGxvdCB3aXRoIHRoZSBhY2N1cmFjeSBvbiB0aGUgeS1heGlzIGFuZCB0aGUgcGVuYWx0eSB0ZXJtIG9uIHRoZSB4LWF4aXMuIFB1dCB0aGUgeC1heGlzIG9uIHRoZSBsb2cgc2NhbGUuICANCiogVXNlIHRoZSBgc2VsZWN0X2Jlc3QoKWAgZnVuY3Rpb24gdG8gZmluZCB0aGUgYmVzdCB0dW5pbmcgcGFyYW1ldGVyLCBmaXQgdGhlIG1vZGVsIHVzaW5nIHRoYXQgdHVuaW5nIHBhcmFtZXRlciB0byB0aGUgZW50aXJlIHRyYWluaW5nIHNldCAoSElOVDogYGZpbmFsaXplX3dvcmtmbG93KClgIGFuZCBgZml0KClgKSwgYW5kIGRpc3BsYXkgdGhlIG1vZGVsIHJlc3VsdHMgdXNpbmcgYHB1bGxfd29ya2Zsb3dfZml0KClgIGFuZCBgdGlkeSgpYC4gQXJlIHRoZXJlIHNvbWUgdmFyaWFibGVzIHdpdGggY29lZmZpY2llbnRzIG9mIDA/DQoNCmBgYHtyfQ0Kc2V0LnNlZWQoNDk0KSAjIGZvciByZXByb2R1Y2liaWxpdHkNCmhvdGVsc19jdiA8LSB2Zm9sZF9jdihob3RlbHNfdHJhaW5pbmcsIHYgPSA1KQ0KDQoNCnBlbl9ncmlkIDwtIGdyaWRfcmVndWxhcihwZW5hbHR5KCksbGV2ZWxzID0gMTApI25lZWRzIGEgbGlzdCBvZiAxMCBwZW5hbHR5IHBhcmFtZXRlcmVzDQpwZW5fZ3JpZA0KDQojIGhvdGVsc19maXRfY3YgPC0NCiMgICAjIFRlbGwgaXQgdGhlIHdvcmtmbG93DQojICAgaG90ZWxzX3dmICU+JSANCiMgICAjIEZpdCB0aGUgbW9kZWwgKHVzaW5nIHRoZSB3b3JrZmxvdykgdG8gdGhlIGN2IGRhdGENCiMgICBmaXRfcmVzYW1wbGVzKGhvdGVsc19jdikNCg0KDQpob3RlbHNfbGFzc29fdHVuZSA8LQ0KICBob3RlbHNfd2YgJT4lIA0KICB0dW5lX2dyaWQoaG90ZWxzX3JlY2lwZSwgcmVzYW1wbGVzID0gaG90ZWxzX2N2LCBncmlkID0gcGVuX2dyaWQpICNpbnN0ZWFkIG9mIGZpdF9yZXNhbXBsZXMoKQ0KaG90ZWxzX2xhc3NvX3R1bmUNCg0KIyBjb2xsZWN0X21ldHJpY3MoaG90ZWxzX3JlcykNCmBgYA0KYGBge3J9DQpjb2xsZWN0X21ldHJpY3MoaG90ZWxzX2xhc3NvX3R1bmUpDQpjb2xsZWN0X21ldHJpY3MoaG90ZWxzX2xhc3NvX3R1bmUpICU+JSANCiAgZ2dwbG90KGFlcyh4ID0gbG9nMTAocGVuYWx0eSksIHk9IG1lYW4sIGNvbG9yID0gLm1ldHJpYykpKw0KICBnZW9tX3BvaW50KCkNCmBgYA0KDQpgYGB7cn0NCmhvdGVsc19iZXN0cGFyYW08LSBzZWxlY3RfYmVzdChob3RlbHNfbGFzc29fdHVuZSwgbWV0cmljID0gJ3JvY19hdWMnKQ0KaG90ZWxzX2Jlc3RwYXJhbQ0KDQpob3RlbHNfZmluX3dmIDwtIGhvdGVsc193ZiAlPiUgDQogIGZpbmFsaXplX3dvcmtmbG93KGhvdGVsc19iZXN0cGFyYW0pDQoNCmhvdGVsc19maW5fd2YNCmBgYA0KYGBge3J9DQpob3RlbHNfZml0IDwtIGhvdGVsc19maW5fd2YgJT4lDQogIGZpdChkYXRhID0gaG90ZWxzX3RyYWluaW5nKQ0KYGBgDQoNCg0KYGBge3J9DQojaG90ZWxzX2ZpdA0KYGBgDQoNCg0KYGBge3J9DQojICNhbHRlcm5hdGl2ZQ0KIyBzZXQuc2VlZCg0OTQpICMgZm9yIHJlcHJvZHVjaWJpbGl0eQ0KIyBob3RlbHNfY3YgPC0gdmZvbGRfY3YoaG90ZWxzX3RyYWluaW5nLCB2ID0gNSkNCiMgcGVuX2dyaWQgPC0gZ3JpZF9yZWd1bGFyKHBlbmFsdHkoKSxsZXZlbHMgPSAxMCkjbmVlZHMgYSBsaXN0IG9mIDEwIHBlbmFsdHkgcGFyYW1ldGVyZXMNCiMgDQojIA0KIyBob3RlbHNfZml0IDwtIGhvdGVsc193ZiAlPiUNCiMgICBhZGRfbW9kZWwoaG90ZWxzX21vZCkgJT4lDQojICAgZml0KGRhdGEgPSBob3RlbHNfdHJhaW5pbmcpDQojIA0KIyBob3RlbHNfZml0ICU+JQ0KIyAgIHB1bGxfd29ya2Zsb3dfZml0KCkgJT4lDQojICAgdGlkeSgpDQpgYGANCg0KDQo3LiBOb3cgdGhhdCB3ZSBoYXZlIGEgbW9kZWwsIGxldCdzIGV2YWx1YXRlIGl0IGEgYml0IG1vcmUuIEFsbCB3ZSBoYXZlIGxvb2tlZCBhdCBzbyBmYXIgaXMgdGhlIGNyb3NzLXZhbGlkYXRlZCBhY2N1cmFjeSBmcm9tIHRoZSBwcmV2aW91cyBzdGVwLiANCg0KKiBDcmVhdGUgYSB2YXJpYWJsZSBpbXBvcnRhbmNlIGdyYXBoLiBXaGljaCB2YXJpYWJsZXMgc2hvdyB1cCBhcyB0aGUgbW9zdCBpbXBvcnRhbnQ/IEFyZSB5b3Ugc3VycHJpc2VkPyAgDQoqIFVzZSB0aGUgYGxhc3RfZml0KClgIGZ1bmN0aW9uIHRvIGZpdCB0aGUgZmluYWwgbW9kZWwgYW5kIHRoZW4gYXBwbHkgaXQgdG8gdGhlIHRlc3RpbmcgZGF0YS4gUmVwb3J0IHRoZSBtZXRyaWNzIGZyb20gdGhlIHRlc3RpbmcgZGF0YSB1c2luZyB0aGUgYGNvbGxldF9tZXRyaWNzKClgIGZ1bmN0aW9uLiBIb3cgZG8gdGhleSBjb21wYXJlIHRvIHRoZSBjcm9zcy12YWxpZGF0ZWQgbWV0cmljcz8NCiogVXNlIHRoZSBgY29sbGVjdF9wcmVkaWN0aW9ucygpYCBmdW5jdGlvbiB0byBmaW5kIHRoZSBwcmVkaWN0ZWQgcHJvYmFiaWxpdGllcyBhbmQgY2xhc3NlcyBmb3IgdGhlIHRlc3QgZGF0YS4gU2F2ZSB0aGlzIHRvIGEgbmV3IGRhdGFzZXQgY2FsbGVkIGBwcmVkc2AuIFRoZW4sIHVzZSB0aGUgYGNvbmZfbWF0KClgIGZ1bmN0aW9uIGZyb20gYGRpYWxzYCAocGFydCBvZiBgdGlkeW1vZGVsc2ApIHRvIGNyZWF0ZSBhIGNvbmZ1c2lvbiBtYXRyaXggc2hvd2luZyB0aGUgcHJlZGljdGVkIGNsYXNzZXMgdnMuIHRoZSB0cnVlIGNsYXNzZXMuIFdoYXQgaXMgdGhlIHRydWUgcG9zaXRpdmUgcmF0ZSAoc2Vuc2l0aXZpdHkpPyBXaGF0IGlzIHRoZSB0cnVlIG5lZ2F0aXZlIHJhdGUgKHNwZWNpZmljaXR5KT8gU2VlIHRoaXMgW1dpa2lwZWRpYV0oaHR0cHM6Ly9lbi53aWtpcGVkaWEub3JnL3dpa2kvQ29uZnVzaW9uX21hdHJpeCkgcmVmZXJlbmNlIGlmIHlvdSAobGlrZSBtZSkgdGVuZCB0byBmb3JnZXQgdGhlc2UgZGVmaW5pdGlvbnMuDQoqIFVzZSB0aGUgYHByZWRzYCBkYXRhc2V0IHlvdSBqdXN0IGNyZWF0ZWQgdG8gY3JlYXRlIGEgZGVuc2l0eSBwbG90IG9mIHRoZSBwcmVkaWN0ZWQgcHJvYmFiaWxpdGllcyBvZiBjYW5jZWxpbmcgKHRoZSB2YXJpYWJsZSBpcyBjYWxsZWQgYC5wcmVkXzFgKSwgZmlsbGluZyBieSBgaXNfY2FuY2VsZWRgLiBVc2UgYW4gYGFscGhhID0gLjVgIGFuZCBgY29sb3IgPSBOQWAgaW4gdGhlIGBnZW9tX2RlbnNpdHkoKWAuIEFuc3dlciB0aGVzZSBxdWVzdGlvbnM6IGEuIFdoYXQgd291bGQgdGhpcyBncmFwaCBsb29rIGxpa2UgZm9yIGEgbW9kZWwgd2l0aCBhbiBhY2N1cmFjeSB0aGF0IHdhcyBjbG9zZSB0byAxPyBiLiBPdXIgcHJlZGljdGlvbnMgYXJlIGNsYXNzaWZpZWQgYXMgY2FuY2VsZWQgaWYgdGhlaXIgcHJlZGljdGVkIHByb2JhYmlsaXR5IG9mIGNhbmNlbGluZyBpcyBncmVhdGVyIHRoYW4gLjUuIElmIHdlIHdhbnRlZCB0byBoYXZlIGEgaGlnaCB0cnVlIHBvc2l0aXZlIHJhdGUsIHNob3VsZCB3ZSBtYWtlIHRoZSBjdXRvZmYgZm9yIHByZWRpY3RlZCBhcyBjYW5jZWxlZCBoaWdoZXIgb3IgbG93ZXIgdGhhbiAuNT8gYy4gV2hhdCBoYXBwZW5zIHRvIHRoZSB0cnVlIG5lZ2F0aXZlIHJhdGUgaWYgd2UgdHJ5IHRvIGdldCBhIGhpZ2hlciB0cnVlIHBvc2l0aXZlIHJhdGU/IA0KDQpgYGB7cn0NCg0KbGlicmFyeSh2aXApDQpob3RlbHNfZml0ICU+JSANCiAgcHVsbF93b3JrZmxvd19maXQoKSAlPiUgDQogIHZpcCgpDQoNCg0KYGBgDQpyZXNlcnZlZF9yb29tX3R5cGUsIGRlcG9zaXRfdHlwZSBhbmQgYXNzaWduZWRfcm9vbV90eXBlIGFwcGVhciB0byBiZSBrZXkgcHJlZGljdG9yIHZhcmlhYmxlcy4gSSBleHBlY3RlZCBkZXBvc2l0X3R5cGUgdG8gYmUgYW4gaW1wb3J0YW50IHByZWRpY3RvciBidXQgdGhlIG90aGVyIDIgcHJlZGljdG9ycyB3aGljaCBhcmUgYXNzb2NpYXRlZCB3aXRoIHJvb21fdHlwZSB3YXMgc3VwcmlzaW5nIHRvIG1lLg0KDQpgYGB7cn0NCmhvdGVsc19maW5fdGVzdCA8LSBob3RlbHNfZmluX3dmICU+JSANCiAgbGFzdF9maXQoaG90ZWxzX3NwbGl0KSANCg0KaG90ZWxzX2Zpbl90ZXN0ICU+JSBjb2xsZWN0X21ldHJpY3MoKQ0KYGBgDQpUaGUgY3Jvc3MgdmFsaWRhdGVkIG1ldHJpY3Mgd2VyZSByb2NfYXVjOjAuODk0IGFuZCBhY2N1cmFjeTogMC44MTQgYW5kIGNvbXBhcmVkIHRvIHRoYXQgdGhlIHRlc3RpbmcgbWV0cmljcyBhcmUgYWJvdXQgdGhlIHNhbWUgd2l0aCB0aGUgYWNjdXJhY3kgdmFsdWUgdmVyeSBzbGlnaHRseSBiZXR0ZXIgdGhhbiB0aGUgY3Jvc3MgdmFsaWRhdGVkIG9uZS4gDQoNCg0KYGBge3J9DQpwcmVkczwtY29sbGVjdF9wcmVkaWN0aW9ucyhob3RlbHNfZmluX3Rlc3QpIA0KDQpob3RlbHNfbWF0PC1wcmVkcyU+JQ0KICBjb25mX21hdChpc19jYW5jZWxlZCwgLnByZWRfY2xhc3MpDQoNCmhvdGVsc19tYXQNCmBgYA0KYGBge3J9DQp0cnVlX3BvcyA8LSA3MDA3DQp0cnVlX25lZyA8LSAxNzMyNA0KZmFsc2VfcG9zPC0gMTYxNA0KZmFsc2VfbmVnPC0gMzkwMQ0KDQpzZW5zaXRpdml0eTwtIHRydWVfcG9zLyh0cnVlX3BvcytmYWxzZV9uZWcpDQpzcGVjaWZpY2l0eTwtIHRydWVfbmVnLyh0cnVlX25lZytmYWxzZV9wb3MpDQoNCnNlbnNpdGl2aXR5DQpzcGVjaWZpY2l0eQ0KYGBgDQoNCg0KYGBge3J9DQpwcmVkcyU+JQ0KICBnZ3Bsb3QoYWVzKHggPSAucHJlZF8xLCBmaWxsID0gaXNfY2FuY2VsZWQpKSsNCiAgZ2VvbV9kZW5zaXR5KGFscGhhID0gMC41LCBjb2xvciA9IE5BKQ0KYGBgDQoNCmEuIFdlIHdvdWxkIGhhdmUgMiBwZWFrcyBhdCAwIGFuZCAxIHdpdGggMSBjb3JyZXNwb25kaW5nIGFuZCB0aGUgbWlkZGxlIHBhcnRzIHdvdWxkIGJlIGNsb3NlIHRvIDAuDQoNCmIuIFdlIGxvd2VyIHRoZSBjdXRvZmYgYmVjYXVzZSB0aGUgc3BlY2lmaWNpdHkgaXMgaGlnaGVyIHRoYW4gc2Vuc2l0aXZpdHkNCg0KYy4gSXQgd2lsbCBnbyBkb3duLCBzaW5jZSB3ZSB3aWxsIGJlIG1ha2luZyBtb3JlIGlzX2NhbmNlbGxlZCA9IHRydWUgcHJlZGljdGlvbnMuIEZhbHNlIHBvc2l0aXZlcyB3b3VsZCBpbmNyZWFzZSBhbmQgc28sIHRydWUgbmVnYXRpdmVzIHdvdWxkIGluY3JlYXNlLiANCg0KOC4gTGV0J3Mgc2F5IHRoYXQgdGhpcyBtb2RlbCBpcyBnb2luZyB0byBiZSBhcHBsaWVkIHRvIGJvb2tpbmdzIDE0IGRheXMgaW4gYWR2YW5jZSBvZiB0aGVpciBhcnJpdmFsIGF0IGVhY2ggaG90ZWwsIGFuZCBzb21lb25lIHdobyB3b3JrcyBmb3IgdGhlIGhvdGVsIHdpbGwgbWFrZSBhIHBob25lIGNhbGwgdG8gdGhlIHBlcnNvbiB3aG8gbWFkZSB0aGUgYm9va2luZy4gRHVyaW5nIHRoaXMgcGhvbmUgY2FsbCwgdGhleSB3aWxsIHRyeSB0byBhc3N1cmUgdGhhdCB0aGUgcGVyc29uIHdpbGwgYmUga2VlcGluZyB0aGVpciByZXNlcnZhdGlvbiBvciB0aGF0IHRoZXkgd2lsbCBiZSBjYW5jZWxpbmcgaW4gd2hpY2ggY2FzZSB0aGV5IGNhbiBkbyB0aGF0IG5vdyBhbmQgc3RpbGwgaGF2ZSB0aW1lIHRvIGZpbGwgdGhlIHJvb20uIEhvdyBzaG91bGQgdGhlIGhvdGVsIGdvIGFib3V0IGRlY2lkaW5nIHdobyB0byBjYWxsPyBIb3cgY291bGQgdGhleSBtZWFzdXJlIHdoZXRoZXIgaXQgd2FzIHdvcnRoIHRoZSBlZmZvcnQgdG8gZG8gdGhlIGNhbGxpbmc/IENhbiB5b3UgdGhpbmsgb2YgYW5vdGhlciB3YXkgdGhleSBtaWdodCB1c2UgdGhlIG1vZGVsPyANCg0KQnkgdGhlIHZhcmlhYmxlIGltcG9ydGFuY2UgcGxvdCB3ZSBkaWQgZWFybGllciwgdGhlIGhvdGVsIHNob3VsZCBwcm9iYWJseSBjYWxsIHBlb3BsZSB3aG8gaGF2ZSByZXNlcnZlZCByb29tIHR5cGUgcCwgbWFkZSBub24gcmVmdW5kYWJsZSBkZXBvc2l0cyBvciBhc3NpZ25lZCByb29tIHR5cGUgSSAoaW4gb3JkZXIgb2YgaW1wb3J0YW5jZSkgc2luY2UgdGhleSBhcmUgbW9yZSBsaWtlbHkgdG8gYmUgc29tZW9uZSB3aG8gaXMgd2lsbGluZyB0byBjYW5jZWwuIA0KDQpUbyBtZWFzdXJlIGlmIGl0IHdhcyB3b3J0aCB0aGUgZWZmb3J0IG9mIGNhbGxpbmcgeW91IGNvdWxkIG1heWJlIGRldmVsb3AgYSBtZXRyaWMgd2hpY2ggaW52b2x2ZXMgdGhlIHByb2JhYmlsaXR5IG9mIGNhbmNlbGxhdGlvbiBvZiB0aGF0IGJvb2tpbmcsIHRoZSBwcmljZSBvZiB0aGUgcm9vbSAoaG93IG11Y2ggdGhlIGhvdGVsIGNvdWxkIHBvc3NpYmx5IHNhdmluZyBieSBtaW5pbWl6aW5nIGNhbmNlbGxhdGlvbnMpIGFuZCB0aGUgd2FpdGluZyBsaXN0IG9mIHRoZSBob3RlbC4NCg0KVGhleSBtaWdodCBiZSBhYmxlIHRvIHVzZSB0aGUgbW9kZWwgdG8gZG8gb3ZlcmJvb2tpbmcgc2ltaWxhciB0byBhaXJsaW5lcyBidXQgdGhpcyBjb3VsZCBiZSBtb3JlIHByb2JsZW1hdGljIHNpbmNlIHRoZXJlIG1pZ2h0IG5vdCBiZSBhIGxhcmdlIHN1cHBseSBvZiBhbHRlcm5hdGl2ZSByb29tcyBpbiB0aGUgYXJlYSBzaW1pbGFyIHRvIGEgbGFyZ2Ugc3VwcGx5IG9mIGFsdGVybmF0aXZlIGZsaWdodHMgdGhhdCBhaXJsaW5lcyBjb3VsZCBwcm92aWRlLg0KDQo5LiBIb3cgbWlnaHQgeW91IGdvIGFib3V0IHF1ZXN0aW9uaW5nIGFuZCBldmFsdWF0aW5nIHRoZSBtb2RlbCBpbiB0ZXJtcyBvZiBmYWlybmVzcz8gQXJlIHRoZXJlIGFueSBxdWVzdGlvbnMgeW91IHdvdWxkIGxpa2UgdG8gYXNrIG9mIHRoZSBwZW9wbGUgd2hvIGNvbGxlY3RlZCB0aGUgZGF0YT8gDQoNCkNvdW50cnkgYmlhcyBjb3VsZCBiZSBzb21ldGhpbmcgdG8gY29uc2lkZXIgc2luY2UgdGhhdCBpcyBhIHZhcmlhYmxlIGluIHRoZSBkYXRhc2V0LiBUaGUgbW9kZWwgY291bGQgZ2l2ZSBxdWVzdGlvbmFibGUgcmVzdWx0cyBkdWUgdG8gaXNzdWVzIHN1Y2ggYXMgcXVlc3Rpb25hYmxlIHNhbXBsZSBzaXplcyBmcm9tIGNlcnRhaW4gY291bnRyaWVzIGFuZCB0aGlzIG1pZ2h0IGxlYWQgdG8gc29tZSBzb3J0IG9mIGRpc2NyaW1pbmF0b3J5IHByYWN0aWNlLiANCg0KWW91IHdvdWxkIGhhdmUgdG8gbWFrZSBzdXJlIHRoYXQgdGhlIGNhbGxpbmcgdG8gY2FuY2VsIGJvb2tpbmdzIGRvIG5vdCBoYXZlIHNvY2lvZWNvbm9taWMgb3IgcmFjaWFsIGJpYXMgaGlkZGVuIGluIHZhcmlhYmxlcyBzdWNoIGFzIHJlc2VydmVkX3Jvb21fdHlwZSBhbmQgZGVwb3NpdF90eXBlLiANCg0KDQoNCg0KDQoNCiMjIEJpYXMgYW5kIEZhaXJuZXNzDQoNCkxpc3RlbiB0byBEci4gUmFjaGVsIFRob21hcydzICBbQmlhcyBhbmQgRmFpcm5lc3MgbGVjdHVyZV0oaHR0cHM6Ly9ldGhpY3MuZmFzdC5haS92aWRlb3MvP2xlc3Nvbj0yKS4gV3JpdGUgYSBicmllZiBwYXJhZ3JhcGggcmVmbGVjdGluZyBvbiBpdC4gWW91IG1pZ2h0IGFsc28gYmUgaW50ZXJlc3RlZCBpbiByZWFkaW5nIHRoZSBbUHJvUHVibGljYSBhcnRpY2xlXShodHRwczovL3d3dy5wcm9wdWJsaWNhLm9yZy9hcnRpY2xlL21hY2hpbmUtYmlhcy1yaXNrLWFzc2Vzc21lbnRzLWluLWNyaW1pbmFsLXNlbnRlbmNpbmcpIERyLiBUaG9tYXMgcmVmZXJlbmNlcyBhYm91dCB1c2luZyBhIHRvb2wgY2FsbGVkIENPTVBBUyB0byBwcmVkaWN0IHJlY2lkaXZpc20uIFNvbWUgcXVlc3Rpb25zL2lkZWFzIHlvdSBtaWdodCBrZWVwIGluIG1pbmQ6DQoNCiogRGlkIHlvdSBoZWFyIGFueXRoaW5nIHRoYXQgc3VycHJpc2VkIHlvdT8gIA0KKiBXaHkgaXMgaXQgaW1wb3J0YW50IHRoYXQgd2UgcGF5IGF0dGVudGlvbiB0byBiaWFzIGFuZCBmYWlybmVzcyB3aGVuIHN0dWR5aW5nIGRhdGEgc2NpZW5jZT8gIA0KKiBJcyB0aGVyZSBhIHR5cGUgb2YgYmlhcyBEci4gVGhvbWFzIGRpc2N1c3NlZCB0aGF0IHdhcyBuZXcgdG8geW91PyBDYW4geW91IHRoaW5rIGFib3V0IHBsYWNlcyB5b3UgaGF2ZSBzZWVuIHRoZXNlIHR5cGVzIG9mIGJpYXNlcz8NCg0KDQo=